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AUGMENTED VIDEO SURVEILLANCE SYSTEM 
BACKGROUND TO THE INVENTION 

THIS invention relates to an augmented video surveillance system. 

Video surveillance or CCTV (closed circuit television) surveillance systems 
usually make use of a number of video cameras and communication links 
to transfer camera-captured analogue images to remote video monitors 
where they can be observed by human operators or to video image 
recordal equipment which records them for subsequent viewing by human 
operators. Such systems are entirely reliant upon the operator's ability to 
discern, from the images viewed, whether a particular event has taken 
place and whether any particular action is required. 

Recognised problems with known video surveillance systems, whether for 
security or other purposes, are boredom and inefficient or improper reaction 
to specified events by surveillance operators. A further problem is that 
there are only limited methods for measuring or monitoring the performance 
of surveillance operators. 

The present invention seeks to address these problems. 
SUMMARY OF THE INVENTION 

According to the invention there is provided an augmented video 
surveillance system in which a surveillance video is augmented by 
insertion, into live video images seen by an operator, of synthetic or 
realistic objects and/or by visual enhancement of predetermined events 
taking place In the live video image seen by the operator, such insertion 
requiring specific action by the operator, and monitoring the action taken by 
the operator in response to such Insertion. 



wo 2005/036881 



-2- 



PCT/IB2004/003304 



The objective of the Invention is to Improve the perfonnance and 
effectiveness of the operator and to measure his performance online. In 
accordance with the Invention this object is addressed by engaging the 
operator's attention by interaction wiVh the live video with the result that the 
operator's levels of vigilance and interest are elevated. The system 
proposed by the Invention may be used to provide on-line operator training. 

The Invention also provides apparatus for use in the system summarized 
above. 

BRIEF DESCRIPTION OF THE DRAWING 

The invention will now be described in more detail, by way of example only, 
with reference to the accompanying block diagram which illustrates the 
principles of the invention. 

SPECIFIC DESCRIPTION 

Referring to the diagram, the numeral 10 indicates an event controller 
which controls and implements video augmentation according to the 
Invention. The event controller 10 can be programmed to augment a live 
video seen by an operator in a number of different ways which may be 
implemented alone or in combination with one another. 

In one version live video events are brought to the attention of the operator. 
These are actual live events taking place in the scene observed by the 
operator. Examples of typical events include entry of a person into the 
observed scene, addition or removal of inanimate objects to the scene. This 
is indicated by the numeral 1 1 in the diagram. 

As described in a simultaneously filed, co-pending patent application filed 
by the present applicant and entitled "Enhanced Video Based Surveillance 
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System" a background image or model Is generated and, during the course 
of live video monitoring, the overall image is segmented into respective 
bacl<ground and foreground segments and video enhancement techniques 
are employed visually to enhance all or selected foreground features or 
events, thereby drawing the operator's attention to such features and 
events. 

The system described above may be interfaced with a video motion 
detection system also described in the aforementioned co-pending patent 
application, in which case the operator's attention is drawn, by camera 
switching and/or image enhancement to specified motion events in the 
observed image. This is indicated by the numeral 13 in the diagram. 

The operator is trained to respond to specified foreground features or 
events, in each case in a predetermined manner. The system may be used 
to measure the operator's efficiency in responding in the appropriate 
manner to specified events. 

The invention proposes the insertion of certain synthetic or realistic objects 
or events into the image viewed by the operator. In one example the event 
controller 10 acts in response to a rules-based event generation module 12 
to initiate the generation, by a synthetic object generator 14, of a synthetic 
event or object and augments the raw. live video image 16 by inserting 
such event or object into the image, thereby forming an augmented video 
Image 18 which is viewed by the operator. In each case the controller 10 
controls all parameters relating to the insertion, e.g. the frequency of 
insertion, the duration of the insertion and so on. 

Synthetic objects are objects which are inserted into the viewed image and 
are classified as synthetic because they do not blend into the imaged scene 
and can be cleariy identified by the operator as artificially inserted objects. 
Typical examples are fanciful shapes such as stars, squares, circles, still 
pictures of persons or objects, or animated sequences of person and/or 
objects. In each case the insertion may require specific action by the 
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operator, as indicated by the numeral 20. The operator may, for example, 
be required to operate a mouse pointer and to click on the insertion In order 
to remove it. The operator's reaction time and ability to carry out the 
required acHon can be monitored to provide an Indication of the operator's 
efficiency and training and can, If necessary, be logged as part of the 
operator's performance record. 

Associated with the insertion of synthetic objects can be certain specific 
tasl<s that need to be performed by the operator, such tasks requiring 
higher levels of cognitive engagement than a simple response of 
acknowledging the presence of the synthetic object by clicking on it For 
example, this could be a discrimination task whereby the operator is 
required to distinguish between different categories of inserted synthetic 
objects. For example, still pictures or Images of persons may show such 
persons displaying normal, unsuspicious behaviour and may require a first 
type of response by the operator. On the other hand, pictures of persons 
displaying suspicious behaviour may require a totally different type of 
response by the operator. Once again, the operator's reaction to the 
inserted object(s) can be used to provide a measure of ability and 
efficiency, with the accuracy of completion of the associated task providing 
an additional level of information on operator performance. 

The object insertion technique described in the preceding paragraph also 
serves a valuable function in on-line training of the operator in that it can 
raise the operator's awareness of certain types of personal behaviour which 
may be considered undesirable from a security point of view, for example a 
person assuming a certain bodily posture in the vicinity of a high-risk area 
such as a cash-counting or gem sorting station. Through these types of 
synthetic objects, operators are sensitised to the types of 
behaviours/postures . that they should detect, thereby increasing the 
likelihood of detecting incidents associated with such behaviours/postures 
when they actually do occur. Often such incidents have a fairty low 
fi'equency of occurrence. 



wo 2005/036881 PCT/IB2004/003304 

-5- 



The invention also envisages the augmentation of the live video by 
insertion of realistic objects or events, under the control of the event 
controller 10 and in accordance with the rules-based event generation 
module 12. In this case, realistic \Adeo objects, such as animated 
sequences extracted from previously recorded Images of the scene under 
surveillance, as indicated by the block 22 in the diagram, are inserted baak 
into the live video image 16 to produce the augmented image 18 in such 3 
manner that the operator is unable to distinguish between artificially 
inserted objects and real objects of the scene. So, for instance, live video 
footage of a person moving in the surveilled scene can be inserted to 
produce the augmented image. 

The operator will once again be required to react in a certain way to tho 
augmented video which he sees. Where, for instance, augmentation 
involves insertion of video footage showing an act of theft, the technique 
can be used to monitor the operator's ability to recognize the act and to 
react accordingly. Thus, where performance on inserted synthetic object(s) 
provides a proxy on operator performance, realistic insertion provide a 
direct measure of operator performance on completely realistic incidents. 
Combining both approaches in operator performance monitoring also 
prevents unrealistic estimates of performance of operator who simply 
concentrate on elevating their performance on inserted synthetic objects at 
the expense of detecting real incidents in the surveillance video. 

In each of the examples described above, the operator will be required to 
interact, in accordance with his training, via an interface generator 24. 
Where a synthetic object has been inserted, the operator may for example 
be required to click once, with a mouse pointer, on an image such as a 
fanciful image or a realistic image of non-threatening type or to click twice 
on an image representing a security threat. The event controller may be 
programmed to give feedback to the operator, via the interface 24, to 
indicate that the correct action has been taken. The inserted object may for 
instance be caused to flash and/or to disappear from the image, thereby 
acknowledging the correctness of the operator's action. 



